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Abstract 

Institutions of higher education are struggling to meet the growing demand for online courses 
and programs, partly because many faculty lack experience teaching online. The eCampus 
Quality Instruction Program (eQIP) is an online faculty development program developed to train 
faculty to design and teach fully online courses. The purpose of this article is to describe the 
eQIP (one institution’s multipronged approach to online faculty development), with a specific 
focus on how the overall success of the program is evaluated using surveys, analytics, and social 
network analysis. Reflections and implications for improving practice are discussed. 
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Introduction 

As online enrollments grew over the last decade, Boise State University was confronted 
with a common problem in higher education: What is the best way to design and develop high- 
quality online courses and support faculty as they teach? High-quality online courses begin and 
end with high quality faculty (Dunlap, 2005; Wilson, Ludwig-Hardman, Thornam, & Dunlap, 
2004); most faculty, though, have never taken nor taught a course online before (see Bacow, 
Bowen, Guthrie, Lack, & Long, 2012; Tipple, 2010; Wolf, 2013). To succeed in the online 
classroom, faculty must possess, among other things, pedagogical, facilitative, instructional, 
social, managerial, assessment, and technical competencies (Baran, Correia, & Thompson, 2011). 
Thus, the challenge institutions face is how to develop these competencies in faculty so that they 
are able to develop and teach high quality online courses. Boise State created the eCampus 
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Quality Instruction Program (eQIP), an online faculty development program, to address this 
challenge. The purpose of this article is to describe the eQIP and the multipronged approach used 
to evaluate the eQIP. 


Literature Review 

Institutions across the country have used multiple approaches to address the problem of 
how to develop and teach high quality online courses. On one end of the continuum are 
universities that essentially leave it up to faculty to decide what courses they teach online and 
how they develop them. On the other end of the continuum are institutions, often with a for- 
profit business model, that have adopted a centralized “enterprise” model in which all online 
course development and training takes place (Hennan, 2012; Lowenthal & White, 2009). Most 
institutions lie somewhere in between these two extremes, offering some type of support for 
online course development and teaching. For instance, the University of Colorado Denver takes a 
decentralized approach in which faculty, for the most part, decide what they want to teach online, 
but may be supported by CU Online (a centralized unit for faculty support) when needed; in 
addition to just-in-time instructional design support, CU Online also offers regular workshops 
and annual events to support faculty developing online courses (Lowenthal & Thomas, 2010b). 
On the other hand, faculty at the University of Central Florida complete a cMOOC (connectivist 
Mass Open Online Course) that prepares them to teach online (Moskal, Thompson, & Futch, 
2015). Institutions like the University of the District of Columbia fully support faculty to develop 
online courses and complete a Quality Matters Standards and Peer Review for online courses to 
improve the quality of their online courses (Britto, Ford, & Wise, 2014; see Meyer, 2013, p.95- 
96 for a list of faculty development programs). There are typically three components common to 
models of support for online learning: (1) using instructional designers to design or support 
faculty as they design courses, (2) training faculty to teach online, and (3) using some type of 
quality control system to evaluate and in turn improve the quality of online courses (e.g., Quality 
Matters or the Online Consortium’s 5 Pillars quality framework). 

Building upon what others have already done, Boise State implemented a program called 
the eCampus Quality Instruction Program (eQIP) to improve the design, development, and 
delivery of online courses. The following paper describes the program Boise State developed and 
the mixed methods approach taken to evaluate and improve the program over time. 


Background: The eCampus Quality Instruction Program (eQIP) 

Boise State University began offering courses online in 1989. Over the years, mirroring 
national trends, enrollments grew. Due to this growth, as well as a desire to develop more online 
programs, the eCampus Center created the eCampus Quality Instruction Program (eQIP) in the 
fall of 2012 to improve how courses are designed and taught online at Boise State. The eQIP 
consists of three distinct components: 

• Course Design: A course design seminar; 

• Quality Assurance: A Quality Matters Peer Review, and 

• Teacher Training: A teaching online seminar. 
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Each component is described briefly in the following paragraphs. 

Course Design Seminar 

To teach faculty how to design online courses and to assist them in the development of 
these courses, eCampus developed a 12-week online course design seminar called the eCampus 
Course Design and Development Seminar or eCD2S for short. Most faculty do not have the 
experience or skillset to design online courses (e.g. Bailey & Card, 2009; Baran, Correia, & 
Thompson, 2011; Marek, 2009; Moar, 2006; Schrum, Burbank, Engle, Chambers, & Glassett, 
2005); faculty are content experts, many of whom have limited coursework or structured 
experiences in teaching and learning online (e.g., Sellani & Harrington, 2002). The eCD2S is 
designed specifically for faculty with no previous experience with online course design. The 
seminar has been iteratively improved over time (see Chen, 2014). The seminar currently 
consists of two distinct phases: 

1. Phase 1-Design Phase (Weeks 1-5): A group of faculty, facilitated by instructional 
designers, work online in a Learning Management System to leam the skills 
necessary for online course design; 

2. Phase 2-Development Phase (Week 6-12): The faculty work one-on-one with 
instructional designers to develop their online courses. 

Over time, it became clear that even after faculty completed the seminar, many desired 
and benefited from additional support when designing future online courses. As a result, a 
shorter, condensed version of the course design seminar called the eCampus Course 
Development Phase or eCD was created. The eCD was designed for faculty who have completed 
the 12-week course design seminar (i.e., eCD2S) at Boise State. The eCD essentially focuses 
only on the development phase (i.e., Phase 2) of the original 12-week seminar. In the eCD, 
during 8 weeks, faculty work one-on-one with an instructional designer, while collaborating with 
colleagues, to develop a turnkey-ready online course (which we refer to as a “Master Course”). 
Collaborating with colleagues and instructional designers helps improve the quality of the online 
courses faculty design, even with faculty who already have experience designing online courses 
(Lowenthal & White, 2009). 

Quality Matters Peer Review 

After faculty develop an online course (in either the eCD2S or the eCD) and teach it for 
one semester, the online course is peer reviewed using the Quality Matters’ framework. Quality 
Matters (QM) is a nationally recognized quality assurance program used to improve the design of 
online courses (MarylandOnline, 2011). Following QM’s peer-review process, each course 
developed in the course design seminars is peer-reviewed by three trained QM faculty reviewers 
(Carter-Cram, 2014; see also MarylandOnline, 2011, for more on the QM peer-review model). 
After a course is peer reviewed, the faculty member who originally developed the course updates 
the course based upon the feedback received through the QM peer-review process. The updated 
course then becomes a “master” online course—that is, one that can be reused and taught by any 
Boise State faculty (see Lowenthal & White, 2009, for more on master courses and centralized 
models of course development). 
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Teaching Online Seminar 

Teaching online is different than teaching face-to-face (Stone & Perumean-Chaney, 2011; 
Xu & Morris, 2007). Without proper training, experienced face-to-face faculty often struggle 
when moving to an online environment (Natriello, 2005). While some universities offer a face- 
to-face workshop on how to teach online, Boise State decided that it would be more authentic 
and convenient to train faculty to teach online in a fully online course. The teaching online 
seminar, called the eCampus Teaching Online Seminar or eTOS, is a 6-week online seminar 
facilitated by full-time faculty (Carter-Cram, 2014)—not instructional designers. In this seminar, 
faculty leam how to teach online, what it feels like to be a part of an online faculty community 
(Brooks, 2010; Schrum et al., 2005), how to transfer evidence-based pedagogical practices 
(Bailey & Card, 2009), and finally what it is like to be an online student (Fein & Logan, 2003). 
This seminar was designed both for faculty who have never taught online as well as faculty with 
years of experience teaching online (see Table 2). 

While the eQIP has been offered for a few years now, this case study focuses on the 
faculty and evaluation strategies used during 2014. In 2014, 51 faculty took part in the course 
design seminars, and 65 faculty took part in the teaching online seminars (see Table 1). 

Table 1 

Number of eQIP participants in 2014 



Spring 

Summer 

Fall 

Total 

12-week Course Design Seminar (eCD2S) 

15 

10 

10 

35 

8-week Course Development Seminar (eCD) 

5 

8 

3 

16 

Teaching Online Seminar (eTOS) 

26 

24 

15 

65 

Quality Matters Peer Reviews (QMPR) 

9 

14 

15 

38 


Method 

Meyer (2013) reviewed a series of published literature in online faculty development and 
criticized seven methodological flaws of the selected research. One of the flaws was that 
outcome measures of past faculty development research overly relies on faculty persons’ 
“honesty and self-understanding” (p. 104) of the training. Unlike many faculty development 
programs that simply depend on a short post-workshop survey—sometimes pejoratively referred 
to as a “smile” sheet—we treat eQIP seminars more like for-credit online courses than snapshot 
workshops and therefore use a mixed-method approach of collecting data to find out what is 
working and what is not working in the program (Creswell, 2008; Creswell & Plano, 2007; 
Greene, 2007). Following standard practices of program evaluation (Stufflebeam & Shinkfield, 
2007), empirical data were collected from participants in the course design seminar (i.e. eCD2S) 
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and the teaching online seminar (i.e., eTOS) to evaluate the success of the eQIP and to make 
infonned improvements of the program. This study was both evaluative and exploratory in 
nature. The following questions guided our inquiry: 

1. How did faculty participate in the cCD2S and eTOS? 

2. What were faculty perceptions of the cCD2S and eTOS? 

3. How did faculty skills and dispositions change over time? 

4. What concerns did faculty have with eCD2S and eTOS? 

More specifically, at regular checkpoints, data from surveys (e.g., module surveys, entrance and 
exit surveys), learning analytics, and social network analysis was collected and analyzed to better 
understand faculty participation, perceptions, skills and dispositions, concerns, and thus, the 
overall effectiveness of the eQIP (see Table 2 for an overview of the data collected). 

Table 2 

Overview of Data Collection 


Research Question 

Seminar 

Data 

RQ1. Faculty Participation 

eCD2S 

Analytics of LMS data & Module Surveys 


eTOS 

Analytics of LMS data & Module Surveys 

RQ2. Faculty Perceptions 

eCD2S 

Exit Surveys 


eTOS 

Exit Surveys 

RQ3. Skills & Dispositions 

eCD2S 

Entrance, Midtenn, & Exit Surveys 


eTOS 

Entrance & Exit Surveys 

RQ4. Faculty Concerns 

eCD2S 

Module Surveys, Midterm & Exit Surveys 


eTOS 

Module Surveys & Exit Surveys 


The quantitative data collected from the surveys was analyzed using descriptive statistics 
to reveal faculty participation, perceptions, dispositions, and concerns; the qualitative data were 
coded in Nvivo 10 using an open-coding method; looking for emergent themes, insights, 
feedback, commonalities, and differences (Merriam, 1998). Using multiple sources of data 
facilitated triangulation and meaning-making in the study (Lowenthal & Leech, 2009). In 
addition to data triangulation that was designed and collected by the first author, a 
methodological triangulation was conducted in turn. The synthesized findings were given 
feedback from the third, fourth, and fifth authors who operated the eQIP to obtain practitioner 
and insider perspectives. Finally, the second author joined the research team and served as the 
external member checker who critically reviewed and evaluated the validity and reliability of 
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interpretations. Direct quotes supporting a finding were identified and labeled using the fonnat of 
seminar name, term, and the data source. 

Result and Discussion 

The results and our reflections are presented in the following section, beginning with an 
analysis of faculty participation in the program, then faculty perceptions of the program, 
followed by how faculty skills and dispositions changed over time, and ending with faculty 
concerns about the program. 

Faculty participation 

Faculty are busier than ever before (Lucas & Murry, 2011). However, simply putting 
faculty development online does not magically make it easier for faculty to participate. Research 
suggests that scheduling conflicts and overall lack of interest prevent online faculty from 
engaging in faculty development (Dailey-Hebert, Mandemach, Donnelli-Sallee, & Norris, 2014). 
Given faculty’s competing priorities and limited time, we contend that it is important for 
institutions, and specifically faculty developers, to analyze how much time faculty are spending 
in online faculty development activities as well as which parts are taking the most (or least) time. 

User behavior 

One way to investigate faculty participation in online faculty development is to analyze 
the analytics in the learning management system (e.g., Vu, Cao, & Cepero, 2014). Two common 
ways to analyze participation using analytics are through looking at hits per page or time logged 
into the system. In 2014, the average page hits per user in the course design seminar ranged from 
70 to 257 hits each week or an average of 132.3 hits across all 6 weeks (see Table 3 and 4). 
When looking at the clicks per week, the first three weeks had more page views than the last 
three weeks (of the design phase). This change is likely due to the content during these weeks; 
weeks 1-3 focused on designing the “skeleton” of an online course, whereas weeks 4-6 focused 
on preparing faculty to develop the course site by outlining corresponding learning activities and 
assessments. The change in page views, however, simply could be the result of time; in other 
words, faculty might simply lose some interest after three weeks into an online faculty 
development seminar, or it could be the result of what else is going on at that time in the 
semester. Additional research needs to be conducted to uncover what might be causing this 
change in page views. Future research like this could help inform faculty developers of the 
optimal length of an online faculty development seminar given the standard ebb and flow (and 
therefore faculty workload) each semester. 

The course design seminar is designed to take faculty 8 hours each week to complete. But 
this seminar is often many faculties’ first experience in an online learning enviromnent. Even 
though the time learners spend in any online course is bound to fluctuate some from week-to- 
week, we strive for these seminars to follow as accurately as possible the time estimates we 
make to faculty when they begin. Therefore, we regularly check the time faculty report they 
spend in these seminars with the time they actually spent logged into Blackboard and then make 
changes when needed. For instance, during the spring of 2014, faculty in the course design 
seminar reported spending 7 to 10 hours a week on the seminar. As a result, we simplified the 
seminar a little by removing some assignments. Then in the following summer and fall, we were 
happy to find that faculty reported spending an average of 5 to 9 hours a week in the seminar. 
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We regularly conduct the same type of analysis on faculty participation in the teaching 
online seminar (eTOS) as we do with the course design seminars. Faculty are expected to spend 
6 hours each week in the 6-week teaching online seminar. Page hits for this seminar ranged 
between 112 and 268 each week. Similar to the frequency of page hits in the course design 
seminar, the average page hits decreased over time. However, whereas time logged into 
Blackboard fluctuated week-to-week in the course design seminar, time logged in the teaching 
online seminar gradually decreased over time (see Table 4). As before, more data is needed to 
understand this gradual decline over time. While findings like these can leave one with more 
questions than answers, analysis like this needs to become a regular practice in online education 
and especially in online faculty development where faculty continue to try to do more with less. 

Table 3 


Comparison of Average Page Hits to Reports of Average Time Spent 



eCD2S 

eTOS 

Average Hits 

Average Time 

Average Hits 

Average Time 

Spring 14 

158.5 

8.5 

154.3 

8.8 

Summer 14 

107.8 

6.9 

164.4 

9.0 

Fall 14 

131.2 

6.3 

168.2 

7.6 


Table 4 


Average Page Views and Time on Task 


4 ± 


vi< J- i i / r r/i i jl ti-u i v 

Course Design Seminar feCD2S) Average Views and Time 



Weekl 

Week2 

Week3 

Week4 

Week5 

Week6 

Spring 

199(8.8hrs) 

173(7.3hrs) 

195.7 (9.5hrs) 

116.1 (8.6hrs) 

127.1 (9.5hrs) 

140 (7.4hrs) 

Summer 

181.5 (9.3hrs) 

83.1 (6.1hrs) 

132(7.6hrs) 

86.1 (6.0hrs) 

93.2 (7. lhrs) 

70.7(5.0hn>) 

Fall 

257.4 (6.9hrs) 

112.6 (5 .Ohrs) 

106.5 (6.8hrs) 

107.5 (5.9hrs) 

94.5(7.0hn>) 

108.6 (6.2hrs) 



Teaching Online Seminar (eTOS) Average Views and Time 



Weekl 

Week2 

Week3 

Week4 

Week5 

Week6 

Spring 

268.8 (9.8hrs) 

259.6 (9.2hrs) 

192.28 (9.0hrs) 

150.37 (9. lhrs) 

132.96 (7.9hrs) 

154.28 (7.9hrs) 

Summer 

234.3 (9.9hrs) 

178.7 (9.3hrs) 

169.4 (9.91ms) 

172(8.5hre) 

140.8 (8.3hrs) 

164.4(8.2hrs) 

Fall 

254.12(10.6hrs) 

203.4 (7.2hrs) 

134(8.1hrs) 

165.9 (62hrs) 

139.8 (6.5hrs) 

112.2(6.8hcs) 
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Discussion participation 

Another way to investigate faculty participation in online faculty development is to 
analyze online discussions (Blignaut & Trollip, 2003). Analyzing the online discussions revealed 
that the facilitators in the course design seminar (who are instructional designers) posted 25% - 
50% of the discussion posts, while the facilitators in the teaching online seminar (who are full¬ 
time faculty) contributed only 8%-30% of the posts (see Table 5). This finding was not a 
surprise; the teaching online seminar is designed intentionally to be less facilitator-driven than 
the course design seminar because the it focuses more on online teaching strategies (Carter-Cram 
& Black, 2014), which among other things, allows faculty to leverage their prior experience as 
educators more than designing online courses from scratch. Further, an intentional goal in the 
teaching online seminar is to help create a community of learners who might assist faculty 
throughout their online teaching careers. 


Table 5 

Facilitator Post Ratio of Seminars 




Course Design Seminar (eCD2S) 




Module 1 

Module 2 

Module 3 

Module 4 

Module 5 

Module 6 

Spring 

26% 

21% 

36% 

31% 

33% 

53% 

Summer 

23% 

30% 

32% 

22% 

39% 

44% 

Fall 

19% 

23% 

20% 

19% 

17% 

19% 



Teaching Online Seminar (eTOS) 




Module 1 

Module 2 

Module 3 

Module 4 

Module 5 

Module 6 

Spring 

21% 

15% 

14% 

17% 

8% 

14% 

Summer 

28% 

23% 

27% 

19% 

25% 

23% 

Fall 

29% 

23% 

25% 

23% 

24% 

26% 


Our faculty new to online education tend to recognize the importance of student-to- 
student interaction and, specifically, the role online discussions play in predominantly 
asynchronous online courses, but they struggle with managing how to facilitate and engage in 
online discussions properly and how they use these discussions in their own online courses. For 
instance, some faculty reported how useful the discussions were in these online seminars because 
they allowed insight into what others were dealing with. One respondent said, “I found many 
were similar to my own issues or ideas ” (eCD2S Spring, module survey), whereas another said 
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that the discussions “ allowed me to ask questions, explore my hypotheses and inclinations and 
see what others have to say, and therefore pennitted the faculty member to “ enjoy the 
collaborative nature of the discussions as well’’ (eTOS Spring, module survey). 

At the same time, others reported that they “struggled to find meaning in the discussion 
boards’’ (eCD2S Fall, exit survey), and “think this is in part because I am an independent 
learner by nature and DB, in general, aren't really my thing (I personally don't feel mandatory 
DB posts aid in my learning of the material, but I do appreciate their role in an online course 
environment)" (eTOS Summer, module survey). 

Analyzing online discussions and specific learners’ perceptions of how their instructors 
use online discussions is important to the success of any discussion-based online course (Ramsay, 
Aman, & Pursel, 2014). Frankly, it takes time and commitment for faculty and students to 
become comfortable and literate with the unique style of reading and writing required to 
participate effectively in electronically mediated environments (Dunlap, Bose, Lowenthal, York, 
Atkinson, & Murtagh, 2016; Vaughan & Garrison, 2005). 

Faculty Perception 

The success of an online faculty development program in an institution where faculty are 
not forced to complete professional development depends heavily on faculty having a positive 
experience and finding value in the program. Therefore, from the inception of eQIP, we wanted 
to know whether faculty were satisfied, feeling a sense of community, and learning new skills 
and dispositions in the program. 

Faculty Satisfaction 

It is common practice to administer an end-of-course evaluation at the end of a workshop 
or course to determine learner satisfaction (Lowenthal, Bauer, & Chen, 2015). However, the 
problem with waiting until the end of the course or workshop to get learner feedback is that once 
the course is over, it is too late to make any changes and possibly improve the experience for the 
cohort learners (Dobrovolny & Lowenthal, 2011). Therefore, after each module in the course 
design and teaching online seminars, faculty are asked to rate their satisfaction on a 5-point 
Likert scale (i.e., Please rate your level of satisfaction with this module/seminar). 

In 2014, faculty reported overall that they were satisfied with the course design and 
teaching online seminars (see Table 6). Faculty responses in the course design seminar ranged 
from 3.55 to 4.78 and from 3.67 to 4.73 in the teaching online seminar. Collecting data after each 
module enabled us to assess the high and low points of each seminar as well as overall trends. 
For instance, a closer look at the teaching online seminar weekly survey results revealed that 
faculty satisfaction gradually increased. Also, while Week 3 in the course design seminar in the 
spring was low regarding faculty satisfaction, in other semesters, faculty reported being satisfied 
with this week, which suggests that there is some value in not making quick changes to a 
workshop simply because of one bad week. Instead, faculty developers should try to collect data 
over time and make informed decisions based on more than one cohort of learners whenever 
possible. 
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Table 6 

Faculty Satisfaction with the Course Design and Teaching Online Seminars 





Course Design Seminar (eCD2S) 





Week 1 

Week 2 

Week 3 

Week 4 

Week 5 

Week 6 

Mid 

Exit 

Spring 

4.00 

4.01 

3.83 

4.38 

4.40 

4.43 

4.50 

3.55 

Summer 

4.00 

4.62 

4.43 

3.71 

4.29 

4.00 

4.43 

4.25 

Fall 

4.75 

4.40 

4.20 

4.71 

4.67 

4.40 

4.57 

4.50 




Teaching Online Seminar (eTOS) 





Week 1 

Week 2 

Week 3 

Week 4 

Week 5 

Week 6 

Mid 

Exit 

Spring 

3.91 

4.18 

4.36 

4.35 

4.40 

4.73 

— 

4.00 

Summer 

4.07 

3.94 

4.19 

4.71 

4.25 

4.28 

— 

4.21 

Fall 

3.67 

4.56 

4.27 

4.80 

4.50 

4.54 

— 

4.34 


While we found it helpful to get a sense of a learner’s overall satisfaction, doing so 
ultimately did not help identify specifically what in a course or workshop faculty are satisfied or 
dissatisfied about. Therefore, we also looked to specific responses from midterm and exit 
surveys to get a better sense of what satisfied our learners. In the course design seminar, faculty 
reported extremely high satisfaction with the instructional designers (who were the facilitators). 
However, not all the faculty were satisfied with the layout of the course. But perhaps most 
interesting is that when faculty were asked whether the seminar was meeting their learning 
expectations, the average satisfaction score decreased from the midterm to the end of the seminar, 
which is when they started working one-on-one with their instructional designers to develop their 
own courses. This decline could be due to the change in the structure of the seminar, from one 
that involved interacting with fellow faculty during the first phase to one where the faculty were 
mostly working alone (with the guidance of an instructional designer) on their own courses. It 
could also be that faculty expected the instructional designers or the development experience 
overall to be something different than it was. Like all online learning experiences, managing the 
expectations of faculty taking part in future seminars might help increase their perceived learning 
during the second half of the seminar. 
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Table 7 

Faculty Perceptions of the Course Design Seminar Spring, Summer, and Fail 2014 


Midterm End 

(N= 33) (N= 25) 


Items 

Mean 

SD 

Mean 

SD 

Content was scholarly-researched and sound. 

4.45 

0.91 

4.29 

0.83 

Content was effective in preparing you to design your online course. 

4.36 

1.06 

4.29 

0.73 

Course layout was easy to follow. 

3.88 

1.22 

4.00 

1.11 

Facilitators demonstrated competence in online education. 

4.75 

0.44 

5 

0 

Facilitators acted respectfully toward participants. 

4.97 

0.18 

5 

0 

Facilitators created frequent interaction and prompt feedback. 

4.63 

0.75 

4.93 

.27 

Seminar was meeting your learning expectations. 

4.47 

0.84 

4.04 

1.18 

Seminar was a professional development community of online 

instructors. 

4.09 

1.06 

4.04 

0.94 


Due to the short duration and already high workload in the teaching online seminar, we 
decided to ask the previous questions only to faculty in the eTOS exit survey. While we lose 
some of the value of the week-to-week data, we recognize that there can come a time where 
learners are surveyed too often. At the end of the teaching online seminar, faculty reported that 
they were satisfied with every item but the course layout (see Table 8). Both online seminars are 
conducted in Blackboard; Blackboard offers faculty the ability to create as many folders as they 
like. While the folders were created with what appeared to be a clear structure and numbering 
system to guide the learners, a few learners reported that it was not as clear as we suspected. This 
data, coupled with some experiments using a screen reader on the seminar, prompted the eQIP 
team to restructure the layout of the teaching online seminar by flattening the entire structure and 
therefore having fewer nested folders. We will continue to collect data to see if this alternation 
changes faculty perceptions of the layout of the seminars over time. 


Online Learning - Volume 21 Issue 1 - March 2017 


95 






Table 8 

Faculty Perceptions of the Teaching Online Seminar During 2014 


Items 

End 

(N=56) 

Mean SD 

Content was scholarly-researched and sound. 

4.38 

1.26 

Content was effective in preparing you to teach your online course. 

4.13 

1.18 

Course layout was easy to follow. 

3.77 

1.04 

Facilitators demonstrated competence in online education. 

4.49 

0.63 

Facilitators acted respectfully toward participants. 

4.73 

0.44 

Facilitators created frequent interaction and prompt feedback. 

4.05 

0.91 

Seminar was meeting your learning expectations. 

4.20 

0.80 

Seminar was a professional development community of online instructors. 

4.38 

0.73 


Perception of faculty learning community 

Research has demonstrated multiple benefits of establishing a community of learners in 
online courses (Abdehnalak, 2015; Boettcher & Conard, 2004; Dzubinski, 2014). Therefore, 
when developing the eQIP, the eQIP team wanted to find a way to create an online faculty 
learning community to both model the power of online community as well as to engage faculty 
further. Community-building activities—influenced in part by the theory and practices of the 
Community of Inquiry (Col) (see Bond, 2011; Dunlap & Lowenthal, 2014; Garrison, Anderson, 
& Archer, 2000; McElrath, & McDowell, 2008; Murdock, & Williams, 2011; Pickett, 2010; 
Stavredes, 2011; Vesely, Bloom, & Sherlock, 2007; Wilcoxon, 2011)—were intentionally 
incorporated into each of these seminars. For instance, a weekly “Dear Jen, Kim, and Patrick” 
letter in the teaching online seminar (see Figure 1) was posted regularly throughout the teaching 
online seminar to model and encourage self-disclosure and a sense of shared connection while 
also leveraging faculty participants’ prior experiences teaching. 

Community is difficult to measure but it begins and ends with interacting with others. 
Therefore, we created weekly social network diagrams using SNAPP (Social Networks Adapting 
Pedagogical Practice, http://www.snappvis.org ) to better understand participants’ interactions in 
the seminar. While student-to-student or student-to-teacher interactions are not unequivocal 
proof of a learning community, they are a necessary building block to establishing and 
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maintaining a community of online learners. Visualizing these interactions (by SNAPP) enabled 
the facilitators to diagnose and monitor participation overtime (see also Dawson, Bakaharia, & 
Heathcote, 2010; McConnick, 2013). These diagrams revealed, for example, that in the teaching 
online seminar, interactions were facilitator-centric (see dark gray node in Figure 7) during the 
first week, but changed to participant-centered over time and eventually began to form what 
appears (when looking at the interactions) as a community of learners (see Figure 8). Supporting 
our theory, a participant in the teaching online seminar noted in the spring of 2014 that “not only 
do I feel happy and excited about my teaching as I exit eTOS, I feel a connection to many more 
professionals at BSU” (eTOS Spring, exit survey). 


Dear Jen, Kim, and Patrick: 

I took your suggestion and had my online students fill out a midterm survey about the course. 
When I looked at the survey results, I felt very discouraged. Students had some good things to 
say about the course, but they also had lots of complaints about the workload, the course 
organization, and the assignments. They even said they felt like the discussion board prompts 
were boring and didn 't inspire meaningful conversation. What should I do now? Should I try to 
change everything they don 't like about the course? Or should I just stick with what I have 

planned and look into making changes after the course is done?" 

Discouraged in Donnelly 

Figure 1. A Sample “Dear Jen, Kim, and Patrick” Letter. 



Figure 2. The Social Network Diagram of Interaction in Teaching Online Seminar (Summer) 


Establishing a learning community takes time (Dzubinski, 2014); unfortunately, though, 
it does not always work as planned (Olson & McCracken, 2015). For instance, in the course 
design seminar, some faculty complained about using the discussion board as a place to “show 
and tell” their design documents—a strategy encouraged by some in the field (see Lowenthal & 
Thomas, 2010a). However, despite resistance from some faculty, other faculty, as evident in the 
quotes below, clearly developed a sense of community over the seminars (largely through their 
interactions in the discussion forums) and in turn considered adding some community building 
activities in their own online courses: 
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• “The sense of community is very important to me and I believe I gained some ground on 
how to promote them in my class” (eCD2S Spring, module survey). 

• “The idea of ‘community’ in an online course is something that I think most of us will have 
to leam and adjust to in a ‘trial by fire’ sort of way” (eCD2S Spring, module survey). 

• “The Discussion Board was a really helpful way to build community and share ideas 
about learning objectives” (eCD2S Fall, module survey). 

• “I read through how the others were approaching this task made me flash on [course 
designs] I hadn’t thought of to try myself “(eCD2S Fall, module survey). 

Changes in faculty attitudes and disposition 

The eQIP is designed to teach faculty how to design high-quality, fully online courses 
from scratch as well as how to teach effectively in this new environment. Hence, we were 
interested in looking at faculty attitudes, knowledge, and dispositions at the end of each seminar, 
as well as how they might have changed over time while participating in the online seminars. For 
the course design seminar, we created a survey with a series of quantitative, Likert scale 
questions as well as qualitative, open-ended questions to measure faculty attitudes and 
dispositions. The survey was administered three times in the course design seminar (i.e., 
beginning, middle, and end of the seminar) and two times in the teaching online seminar (i.e., the 
beginning and end). The results from these surveys are summarized in Table 9 and discussed 
along with open-ended feedback in the following sections. 

Table 9 


Faculty Confidence Designing and Teaching Online Courses During eQIP Seminars in 2014 


Items 

(1 =Strongly Disagree', 5=Strongly Agree ) 


Begin 

(eCD2S 

N=31; 

eTOS 

N=70) 

Midterm 

(eCD2S 

N=33; 

eTOS=n/a) 

Final 

(eCD2S 

N=25; 

eTOS 

N=56) 

1 . 

You are confident in identifying and describing 

eCD2S 

3.51 

3.99 

3.96 


target online student characteristics. 

eTOS 

3.59 

- 

4.28 

2. 

You are confident in analyzing differences 
oeiween me online learning environment ana me 

eCD2S 

3.56 

4.04 

4.18 


on-campus classroom. 

eTOS 

3.78 

— 

4.35 

3. 

You are confident in designing online assessment 

eCD2S 

3.39 

4.16 

4.39 


linked to course learning objectives and 
activities. 

eTOS 

3.93 

- 

4.28 

4. 

You anticipate the course evaluations for your 

eCD2S 

2.64 

— 

2.75 


online course compared to traditional cours es 
will be higher. 

eTOS 

4.82 

- 

3.30 

5. 

You expect the workload to design and develop 

eCD2S 

3.83 

4.36 

4.32 


an online course compared to traditional on e w ill 
be higher. 

eTOS 

4.10 

- 

4.12 

6. 

You expect motivating students in an online 

eCD2S 

3.27 

3.81 

3.89 


course compared to traditional one will be more 
difficult. 

eTOS 

3.28 

- 

3.32 
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Faculty confidence 

We were interested in how faculty confidence in designing and teaching online changed 
over time. We hoped that the eQIP would transfonn participants’ view of online learning from 
content-oriented toward student-oriented (Vries, Grift, &, Jansen, 2014) as well as leave them 
with a more realistic expectation of online teaching. Table 9 shows that faculty confidence in 
identifying and describing target online student characteristics, analyzing differences between 
the online learning environment and the on-campus classroom, and designing online assessments 
linked to course learning objectives and activities increased over time in both seminars. 
Interestingly, participants in these online seminars expressed different attitudes in questions 4, 5, 
and 6. For instance, after faculty participated in the teaching online seminar, they anticipated 
lower course evaluation scores when teaching online courses; a decrease of -1.52 from the 
beginning to the end of the seminar (see Question 4 in Table 9). This result could be due to the 
fact that faculty participants read the assigned textbook about online teaching, realized the 
demands of highly interactive online courses in action, and read research on teaching evaluations 
decreasing when teaching online compared to teaching face-to-face courses (see Lowenthal et ah, 
2015). Then when asked if they expected the workload of designing and teaching to be higher in 
online courses than in face-to-face courses, faculty responses increased over the duration of both 
seminars—meaning that while they expected the workload to be higher, this expectation grew by 
the end of both seminars. However, faculty responses regarding expected workload in the 
teaching online seminar barely increased from the beginning to the end of the seminar, whereas 
faculty response in the course design seminar increased 0.49 by the end of the seminar (see 
Question 5 in Table 9). Finally, when asked about how difficult it will be to motivate online 
students, faculty responses in the teaching seminar decreased some (suggesting that they might 
be gaining confidence in motivating online learners), whereas faculty responses in the course 
design seminar increased over time (+0.62). While it is nonnal to lose confidence once one 
leams more about something (e.g., teacher preparation: students often report being more unsure 
about teaching after they have student taught), a goal of the eQIP is to develop our learners’ 
confidence. Therefore, we will continue to find ways to improve these aspects of our seminars 
over time. 

Perceived changes in online course development 

Teaching is an individual activity; as such, teachers at all levels of education often feel 
isolated (McQuiggan, 2011). Some faculty thrive on this autonomy, while others do not. 
Instructional design, on the other hand (at least in large departments), is often a collaborative 
endeavor; therefore, we intentionally developed the course design seminar to help faculty see 
course design not as an individual intellectual activity (as they naturally might) but as an open 
collaborative activity. Open-ended feedback revealed how faculty sensed their growth. For 
instance, one faculty member reported how the seminar activities provided an opportunity to 
“think about the different scales (course vs. unit/module) of objectives. It helps to be forced to 
understand and articulate how they are connected with one another” (eCD2S Fall, module 
survey). Another reported how figuring “out how I would write directions for an online activity 
was a challenge and eye opener” (eCD2S Fall, module survey). Faculty also reported how these 
seminars helped them rethink how they teach other courses—a goal for nearly any faculty 
developers. For instance, one faculty member reported how she or he now saw the potential to 
“redesign course assessment/evaluation in my other courses, and I'm already telling colleagues 
outside of this seminar about it. Good stuff!” (eCD2S Fall, module survey). 
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Perceived changes in online teaching 

Faculty who have never taught online before often hold certain assumptions about 
teaching online. These can range from “teaching online entails more work than teaching face-to- 
face” to other opinions such as “students cheat more online.” The teaching online seminar is 
designed not only to teach faculty how to teach online but to challenge common misconceptions 
faculty might hold. Participants reported changes in their perception regarding the ability to 
effectively communicate online. For instance, one participant found “orientation videos to be 
personal, infonnative, and a great way to begin an online course,” and that he or she “enjoyed 
watching the samples and getting ideas for [her] own” (eTOS Spring, module survey). Others 
discovered some of the affordances of a LMS. For instance, one participant pointed out how “the 
possibilities for reflection in a safe and confidential modality is important. It's different than the 
discussion board because the public aspect of that really forces you to be on your guard” (eTOS 
Spring, module survey). Finally, others specifically identified how their perceptions of online 
teaching changed but remained realistic. For instance, one participant stated that “I kn ow 
teaching my first online class will take a lot of time; I appreciate the reminders and [facilitator's 
comment to not freak me out” (eTOS Summer, module survey). Perceived changes like these are 
something we hope to see (e.g. Scagnoli, Buki, & Johnson, 2009) from participants as a result of 
taking part in the eQIP. 

Faculty Concerns 

Exit survey results suggest that overall faculty considered the eQIP a valuable faculty 
development experience. However, our inquiry also uncovered concerns faculty had with the 
eQIP as well as online teaching in general. 

Individualized vs. grouped training 

Boise State, like many institutions, is interested in creating new online courses and 
programs to meet student demand as efficiently and inexpensively as possible. The eQIP was 
designed to train faculty to be able not only to design online courses (that they or others would 
teach) but to leam how to teach any online course (whether one they designed or not). Some 
faculty, though, had concerns with this model and felt that it was too standardized. Regarding the 
course design seminar, some faculty reported concerns with standardized course development. 
One faculty member explained that a standardized model of course development “takes longer to 
work within someone else's templates than your own” (eCD2S Summer, midterm survey). Some 
faculty explained that the eQIP-grouped “standardized” training approach does not work well for 
faculty, many of which work better individually (e.g., with a dedicated instructional designer) 
than in groups. One faculty participant captured this sentiment when he or she suggested that 
eCampus should embrace “the idea that faculty learners are very different from graduate students 
and that a personalized, one-on-one training defined by faculty priorities and focusing on 
personalized interaction would be much more effective” (eTOS Spring, exit survey). While we 
recognize that some faculty would prefer a one-on-one approach, there are not enough resources 
to accommodate such an approach. We are, however, continually reminding faculty that they are 
the subject matter experts, that these are their courses, and that they can change the course 
template when needed. 


Online Learning - Volume 21 Issue 1 - March 2017 


100 



Novice vs. veteran faculty participants 

The target audiences of the eQIP are novice online instructors who have little or no prior 
experience designing/developing online courses and/or teaching online at Boise State. However, 
more and more faculty with prior experience as either online students or online instructors at 
prior institutions sign up for the eQIP; they sign up because their specific department expects all 
new online faculty to go through the eQIP. For instance, one faculty participant reported: 

I am beginning to think I am in the wrong course.... I have been trained in course 
development at the graduate level and have participated in accreditation self-studies. 
What I really needed was the skills in the technology of online learning, but you can't 
take that class until you are teaching online for Boise and you can't teach online for 
Boise until you take this course. (eCD2S Spring, module survey). 

Not all faculty who take part in the eQIP with prior online teaching experience are upset. A 
number of them are excited to leam new ways of designing and/or teaching online courses. But 
the majority of these faculty do want an advanced version of the eQIP as captured in the 
following quotes: 

I would encourage you to consider breaking eTOS into two seminars: (1) with standard 
topics for instructors who have never taught online before, much like this seminar; (2) 
built around grouping experienced faculty with shared priorities (collect a ranked set of 
priorities) into small task groups to address those priorities, allowing for extensive one- 
on-one interaction (for example one hour per week via Google Hangouts) with 
instructional design consultants who have expertise in that area (eTOS Spring, exit 
survey). 

Maybe there should be a third option of course development where teachers have taught 
online before, but want to radically alter the design and look of their online course 
(eCD2S Spring, module survey). 

One idea here might be to create a diversion plan - a set of assignments for faculty who 
have never taught online, and a different role set for faculty who have - so that the latter 
are being pushed both to review what they have (rather than plan it) and also to offer 
reflections on what works and not for those who have not. In other words, faculty clearly 
have very varied backgrounds and one size does not fit all (eTOS Spring, module survey). 

Feedback like this argues for a more flexible and individualized and/or customized version of the 
eQIP (eCD2S Review Meeting, August 8, 2014). This complaint is a common problem with 
many online courses, not just online faculty development. The eQIP team continues to consider 
ways to change the eQIP by incorporating individualized learning designs such as worked 
examples (eCD2S Review Meeting, June 4, 2014), quest-based learning and digital badges 
(Haskell, 2013) or competency-based learning (Sally & Louis, 2014). 

Technology as prerequisite vs. just-in-time technical training 

Faculty are subject-matter-experts—not necessarily technology experts (Kukulska-Hulme, 
2012). At Boise State a tension exists, as it does at many other universities, about how many 
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technology skills faculty should be expected to have (or develop) to teach online and where they 
should go to get help when they need it. The eQIP currently does not focus much on developing 
basic technical skills in faculty (e.g., uploading a document to Blackboard). Instead, the 
eCampus takes the stance that all faculty who wish to teach online (and participate in the eQIP) 
should have basic technical competences before beginning the eQIP. However, in practice, 
faculty do not always pay attention to this prerequisite; neither do they understand what basic 
technical skills are in the first place. Thus, we find a reoccurring concern with some faculty is 
about whether being proficient with basic technology should be a prerequisite or whether we 
should provide just-in-time technical training. For instance, one faculty participant suggested that 
there needs “to be better support during the heavy tech portions of the class. ” Along the same 
lines, another person pointed out the need for a “specific contact person [to] be identified to help 
troubleshoot issues” (eTOS Fall, exit survey). Aware of this continuing concern, the eQIP 
facilitators have tried to provide more-than-expected just-in-time technical training (such as tip 
sheets and tutorial videos) to compensate for the gaps in faculty technical skills. And while some 
faculty have praised this flexibility and support—for example, one reported how he or she 
“enjoyed specific skill set instruction” (Camtasia, Google Hangouts, tech skills, etc.) (eTOS 
Summer, exit survey)—the tension still remains about whether it is fair to expect faculty to have 
basic technical skills before teaching online for the first time. 

Experiencing learning as an online student 

Each of the eQIP seminars is conducted fully online, which allows faculty to experience 
firsthand what it is like to be an online student and to experience some of the common 
complications students face (e. g., technical difficulties with Blackboard LMS, using e-textbooks 
and multimedia, academic honesty, accessibility, copyright issues, student interaction, and time 
management [see Mason et al., 2010]). Faculty regularly report that this hidden curriculum is a 
valuable experience. For instance, faculty often recognize how a busy life can get in the way of 
completing an online course. One faculty person said, “I have more sympathy and empathy with 
my online students now. I get to this class after family and work responsibilities; I am trying to 
do enough to get by” (eTOS Spring, exit survey). Another group found themselves questioning 
whether they really should have deadlines over the weekends if they do not want to work over 
the weekends. For instance, one faculty participant in the eCD2S commented: “I am curious as to 
why there are Saturday deadlines and I have been considering whether I want those in my course ” 
(eCD2S Fall, module survey). Additionally, faculty often experience technical problems for the 
first time from a student’s perspective as captured in this quote: “I know this is more of a Bb 
issue but now I understand when my students say - ‘Oh I didn't see that.’ ” (eTOS Fall, module 
survey). 

This authentic experience, though, takes time. Some faculty reported that this authentic 
experience could be done in less time or with less work or that their participation in the seminar 
should not be graded. This concern about being treated too much like an online student was 
addressed in ways in the course design seminar by explaining to faculty that by experiencing 
discussions, assignments, quizzes, and group tools from a student’s view, they would be in a 
better position to resolve students’ issues during teaching. The eTOS facilitators, however, tried 
to make participants’ comments and concerns teachable moments. For instance, after receiving a 
complaint asking “Why am I ‘losing points’ on a self-assessment for not doing an optional 
assignment? That doesn't seem fair!” (eTOS Spring, module survey), facilitators apologized for 
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the error, which was simply an oversight, and explained that errors like this sometimes happen 
when adjustments are made to a course from semester to semester and that faculty should strive 
to double check each course when making even minor adjustments (eTOS Review Meeting, 
October 22, 2014). Such modeling was effective because, as one faculty member reflected in his 

journal entry: “Don't be afraid to admit a mistake. I'm somewhat dyslexic and sometimes 

things just get confused. I tell my students that if you find something that seems odd, just tell me 
because it probably is and I’ll fix it” (eTOS Spring, journal activity). 

Workload and time competition 

Time is a sensitive issue with faculty (Lowenthal, Wray, Bates, Switzer, & Stevens, 
2013). One of the challenges the eQIP faces is competing with limited faculty time. For instance, 
even though the teaching online seminar only takes 6 hours a week for 6 weeks (the shortest of 
the seminars), this is still 6 hours a week that faculty have to take away from teaching, 
scholarship, or service. Despite our best efforts, some participants reported that tasks in seminars 
were “busy work.” For instance, one faculty reported in the teaching online seminar “I simply, 
realistically, literally(!) did not have enough time to complete while at the same time maintain 
my teaching, research, and personal life!!! (Wheww!)” (eTOS Summer, exit survey). This 
tension gets even worse with the course design seminar that takes even more time. Boise State 
faculty, like online faculty elsewhere, regularly report that designing an online course takes more 
time than preparing for a face-to-face course (see also Bento, 2011). However, some qualitative 
data suggest that many of these participants begin to change their perception of “busy work” by 
the end of the seminars. A common response from faculty looks like this: “The material in this 
course was overwhelming many times. At the end I can see that the structure was appropriate” 
(eCD2S Spring, exit survey). This change in perception is even more common in the online 
seminars because early in the course design seminar faculty are required to complete multiple 
worksheets as part of the instructional design process that they are not used to completing when 
designing face-to-face courses. It is usually not until the fifth or sixth week that faculty report 
that they “feel like things are starting to come together in a way that will allow me to best utilize 
my time with the design consultant” (eCD2S Spring, module survey). We have found that faculty 
who sign up for the course design seminar to design a course that they are signed up to teach the 
following semester do better than those who have a few semesters before they actually teach the 
course they are developing. We also began adding time estimates for each activity in the LMS 
for both seminars to help faculty manage their own time. However, providing accurate time 
estimations for all learners is nearly impossible; therefore, some faculty who worked slower than 
others actually were bothered when time estimates did not match the time it took them to 
complete a project. 


Conclusion 

A successful online faculty development program must include pedagogical support, 
technology support, and design and development support (Baran & Correia, 2014) that overcome 
obstacles about time, expertise, and motivation of faculty (Henning, 2012). This study addressed 
the implementation of an online faculty development program at a mid-size, metropolitan 
research university. We investigated faculty participation and perceptions by user pattern, 
discussion participation, satisfaction, learning community, attitude and disposition, as well as 
their concerns. The results of our inquiry suggest that while there are still ways we can improve 
each seminar, overall, the eQIP is preparing Boise State faculty to design, develop, and teach 
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online courses. In 2014, in fact, 51 online courses were developed as a result of the eQIP, 100 
faculty members completed both seminars, and 38 courses went through QMPR. With the 
faculty intent to change teaching practice, eQIP impacts both online and face-to-face teaching. In 
the words of one faculty participant: 

This course is a worthy endeavor for both online and face to face teachers. I will use the 
materials that I made for the online course in face-to-face classes as well. I plan to 
incorporate online components into my face-to-face classes in the future (eCD2S Spring, 
exit survey). 

Future improvement based on lessons learned 

Online faculty development is a rapidly changing field and its offerings require 
continuous evolutions (Meyer & Murrell, 2014). Based on the findings of this study, the 
eCampus Center should continue improving the seminars to be as engaging as possible. Our 
continued data collection efforts reveal the following four potential areas of improvement. 

1. Provide individualized seminar processes 

Several faculty comments indicated that having all participants go through the faculty 
development seminars at the same speed failed to serve faculty from various disciplinary 
backgrounds. A more directional and customized faculty development process may better 
meet faculty’s needs. For example, a discipline-based repository for online course will be 
helpful and flexible for faculty who are not beginners in online education. A faculty 
development program like the eQIP does not have to be 100% online to be authentic, 
relevant, motivating, engaging, and useful for faculty (Sorinola, Thistlethwaite, Davies, 
& Peile, 2015). 

2. Empower faculty ownership of the seminars 

Experienced faculty should be identified as soon as possible and encouraged to become 
mentors to novices in the seminar (Xu & Morris, 2007). Seminar topics should be chosen 
from faculty’s teaching experience to promote faculty ownership of the seminars. 

3. Condense course content 

In spring 2014, the eTOS team invited a new facilitator to lead the seminar. His external 
perspective helped reorganize and remove any redundant infonnation. This significantly 
lowered faculty’s workload in the teaching online seminar. 

4. Provide tenure-related incentives 

The eCampus Center should negotiate at the University level to provide faculty supports 
and recognitions in online teaching. For example, to include the eQIP participation as one 
element in faculty promotion, allow online course developers to count development time 
toward semester teaching loads, and create online teaching fellow programs to recognize 
faculty contribution to online education. 

Limitations 

The purpose of this study was to describe one institution’s effort to develop high-quality 
online courses and high-quality online teachers. Therefore, as a case study within a highly- 
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individualized context, the results of this study offer insights to peer institutions with similar 
backgrounds, but may not be generalizable to a larger population (Stake, 1995; Yin, 2009). But 
even as a case study, the results could have been strengthened by including additional data such 
as a faculty focus group; a design of pre- and post-seminar faculty survey, participants’ retention 
rates, and student outcomes in the high-quality courses developed, and faculty end-of-course 
evaluations teaching these courses. In the future, concerted and coordinated research efforts 
across peer institutions may advance the study of faculty development beyond the level of 
sharing best practices in the field (Meyer, 2013). 
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